Dynamically displaying multiple virtual and augmented reality views on a single display

ABSTRACT

One variation of a method for dynamically displaying multiple virtual and augmented reality scenes on a single display includes determining a set of global transform parameters from a combination of user-defined inputs, user-measured inputs, and device orientation and position derived from sensor inputs; calculating a projection from a configurable function of the global transform parameters, context provided by the user and context specific to a virtual and augmented reality scene; rendering a virtual and augmented reality scene with the calculated projection on a subframe of the display; and repeating the previous two steps to render additional virtual and augmented reality scenes.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.14/044,287, filed Oct. 2, 2013 and claims the benefit of U.S.Provisional Application Ser. No. 61/709,141, filed on Oct. 2, 2012. Thecontents of the foregoing applications are hereby incorporated byreference in their entirety.

TECHNICAL FIELD

This invention relates generally to the virtual and augmented realityfield, and more specifically to a new and useful system and method fordynamically displaying multiple virtual and augmented reality scenes ona single display in the virtual and augmented reality field.

BACKGROUND

With the gaining popularity of mobile phones and mobile/tabletcomputers, virtual and augmented reality applications are becoming moreapproachable by the general public. However, as augmented and virtualrealities become more widely accepted, there are numerous challenges tocreate and interface understandable and navigable by a wide variety ofusers. One aspect of virtual and augmented reality scenes is that it isdifficult to represent the contents of a virtual and augmented realityscenes in a static display because a user's natural experience of a realscene is only in one direction with a fixed field of view. When multiplevirtual and augmented reality scenes are present on a single display,the amount of visual information viewable in each scene is furtherlimited. Thus, there is a need in the virtual and augmented realityfield to create a new and useful system and method for dynamicallydisplaying multiple virtual and augmented reality scenes on a singledisplay. This invention provides such a new and useful system andmethod.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic representation of a method of a first preferredembodiment of the invention; and

FIG. 2 is a representation of using the scene frame as a function to theprojection calculation function.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the inventionis not intended to limit the invention to these preferred embodiments,but rather to enable any person skilled in the art to make and use thisinvention.

As shown in FIG. 1, the method of a preferred embodiment includes thesteps of detecting orientation and position of a computing device ifpossible S110, optionally determining a user's viewing location S172,determining a user-configured transform S120 through either mouse inputsS121A or touch gestures S121B, combining all transforms into a set ofglobal transform parameters consisting of an orientation, a set ofperspective projection parameter transforms and a viewer's locationrelative to the device shared across all VAR scenes S130, collatingrequested VAR scenes into a sequence to be drawn S140, checking whetherthe resources required for each VAR scene has been loaded S150 and ifnot enqueuing the resources to load on a background thread S160B, thenchecking whether the requested frame of the VAR scene is visible onscreen S170 and if so calculating the projection of the VAR scene usinga modular function S170 which can take as inputs a calculation of theVAR scene subframe on the display S171, a set of VAR scene-specificcontext S173, and any other additional global context S174. Once aprojection has been calculated, the VAR scene is rendered to therequested frame S180. Separately, a concurrent method waits for aresource load request S210 and then loads the requested resource S220.In the situation where memory becomes constrained, an additional methodcollects requested VAR scenes S310, checks for which VAR scene resourceshave been loaded S320 and optionally whether the VAR scene is notvisible within the device frame S330 and then evicts the resource frommemory S340.

The virtual and augmented reality scene may consist of a 3D model, apartially or wholly spherical photo, digitally generated environmentmap, or other suitable model presentable in 3D space.

Step S110, which includes detecting orientation of a computing devicefunctions to determine the orientation of the computing device. Thecomputing device preferably includes an inertial measurement unit (IMU)that preferably includes a 3-axis magnetometer, a 3-axis accelerometer,or a 3-axis gyroscope. The IMU may alternatively include any suitablecombination of the above components or a single or double axis sensor orany suitable sensor combination to detect orientation. Additionally,Step S110 may include detecting positioning location, which preferablyprovides another parameter for determining orientation and position. Aglobal positioning system (GPS) can preferably be used to determinegeographical location. Orientation information may be gathered in anysuitable manner including device API's or through any suitable APIexposing device orientation information such as using HTML5 to accessdevice orientation or CoreMotion on an iOS platform or the equivalent onAndroid and Windows platforms. An imaging system may additionally beused to perform image processing on the surrounding environment.Preferably the image system is a forward or backward facing camera thatcan be used to determine the position of a user looking at the computingdevice. In the case where there are no suitable sensors, this step mayrepresent the device orientation as the identity orientation.Optionally, device orientation can be disabled and Step S110 simplyreturns the identity orientation.

Step S120, which includes configuring user-transforms functions to allowthe user to look at a specific portion of the virtual and augmentedreality scene. This potentially includes combining multiple orientationsfrom separate user inputs in whatever mathematical construction ischosen to represent orientation, including, but not limited to: rotationmatrixes, euler angles (yaw, pitch, and roll) and quaternions. Incombining these orientations an operator is applied in a specific order,typically, but not limited to, multiplication. Separately a set oftransforms on the parameters of a perspective projection (including, butnot limited to, field of view, aspect ratio, near plane, far plane, andfrustum bounds (left, right, top, and bottom)) are maintained andmodified by the user and then passed to step S130. Combining thesetransforms may use any suitable mathematical operator such as, but notlimited to, multiplication, division, addition, and subtraction.

Step 121A functions to allow the user direct which direction the user islooking with a mouse cursor. Step 121A maps cursor movement torotations. One preferable mapping is one in which that movement alongthe X-axis of the display introduces an orientation that ispremultiplied with the orientation from step S110 in Step S130,representing rotation in yaw. Movement along the Y-axis of the displaycreates an additional orientation that is pre-multiplied with theorientation from S110 in Step S130, but only after the X-axisorientation is premultiplied, representing rotating in pitch. Preferablythe rotation is mapped monotonically with movement along each access,including, but not limited to, a simple linear relationship. The boundsof this function can be either the frame of the device, the frame of aspecific virtual and augmented reality scene, a frame specified by theuser, or infinite in the case where the user interface paradigm allowspointer lock. Additional mouse inputs such as scrolling may be used totransform parameters of the perspective projections such as, but notlimited to, field of view.

Step 121B, which includes configuring user-transform (consisting of anorientation and optionally a projection transform and optionally apositional offset) using touch gestures, functions to allow the user tomanually position the VAR scene on a touch-screen device such as aphone, tablet or other handheld computer. One such gesture that may beused is a dragging gesture such that an orientation is calculated as ifthe user is looking from the center of a sphere out and dragging thesphere with the center of the sphere acting as a pivot point. In thecase where the VAR scene is a spherical photograph, approximately thesame point in the image should remain under the user's finger throughthe gesture. One such way to calculate such an orientation is to mapeach touch point to a vector out of the center of the screen along anorthonormal basis aligned with the existing global orientation providedin step S130. One preferable method of achieving this is calculating avector in device space and then multiplying it with the globalorientation provided in step S130. A preferable method of estimating thetouch vector in device space, includes but not is not limited to, firstcalculating the x and y coordinates on screen either relative to theentire device frame or relative to a specific VAR scene frame and thencombined with a field of view in a VAR scene is used to calculate x andy coordinates in normalized device coordinates, from which z can bederived as the magnitude of the orientation vector must be one. As theuser moves their finger from the start point, the touch vector can berecalculated and a rotation can be derived between the start and currentvector through typical dot and cross products or any other suitablemathematical construction. In another implementation, the user can beconstrained to only allow rotations about the Z-axis by projecting thetouch vectors onto the horizon plane (at Z=0) before calculating therotation. Alternatively, one may switch the orientation from a drag topush model by simply reversing the orientation derived from the previousstep calculated by determining the start and end touch vectors.

An additional gesture which may be included is one such that somesubframe of the screen is mapped to yaw and pitch rotations similar toS121A, where instead of the cursor location defining the yaw and pitch,the touch event location is used instead.

An additional gesture which may be included is one such that the vectoralong which a user drags their finger can be interpreted as a vector inthe orthonormal basis of the globally transformed orientation. Arotation can then be calculated that rotated the z-axis of the globallytransformed orientation to align with the vector of the drag. This hasthe effect of aligning the horizon perpendicular to the direction of thedragged finger, which can be used to realign gravity in a more usefuldirection when the primary viewing direction is, for example, down in areal-world orthonormal basis.

An additional gesture which may be included is one such that as the userpinches to zoom the field of view property of a perspective transform isscaled monotonically.

An additional gesture which may be include is one in which doubletapping decreases the field of view property of a perspective transform.

Any additional gestures not mentioned here that modify the orientationof VAR scene or parameters of a perspective projection are suitable forStep S121B.

Step S172 provides a method to optionally track the user's locationrelative to the device and display so that the in step S170, the userdefined function can adapt the projection of the VAR scene so simulatethe view through the VAR scene frame to the user's eye as if the VARscene was behind the VAR scene frame in real life. Methods to do thisinclude, but are not limited to head tracking, IR tracking, eyetracking, and using proximity sensors. The user's location can be storedin displacement from the device, absolute world coordinates, or anyother suitable reference frame and coordinate system.

Step S130 provides a method to formulate the transforms provided inS110, S120 and S172 into a data structure that can be used as aparameter in the function used in Step S170 to calculate the individualperspective transforms for each VAR scene. Preferably this involves, butis not limited to, multiplying like transforms together, such asorientation and field of view scale such that the final set ofparameters represents a single orientation, set of perspective transformparameters and user displacement relative to the device.

Step S140 involves collating all requested VAR scenes into a collectionwhich functions as a draw list to the system graphics APIs. Onepreferred method involves, but is not limited to a global object inwhich the VAR scenes register with upon initialization. This may beimplemented as a singleton or just as an object that is passed aroundthe running execution context. Typically each VAR scene is representedas a view in each platform specific graphics API, for example, but notlimited to a UlView subclass on iOS, a div or canvas element in HTML,and the corresponding technologies on Microsoft and Linux platforms.

Step S150 involves determining whether the VAR scene would be visible tothe user. One preferred method involves checking whether the view'sbounds are within the device's bounds, checking whether the view is partof the graphical interface at all, checking whether the view is hidden,checking whether the view is fully transparent, and checking whether theview would be fully obscured by other graphical elements. If the VARscene is not visible to the user, then steps S160A, S170, and S180needed to render the scene are skipped and the next VAR scene isprocessed.

Step S160A involves determining whether the resources required to renderthe VAR scene have been loaded into memory. One possible method is tomaintain a set of resources for each individual VAR scene independently.A preferred method, however, is to maintain a shared set of resourcesfor a group of VAR scenes. This minimizes redundancy as two separatescenes that require the same resources only require one instance of theresources rather than two. Additionally, this improves performance as itminimizes expensive context switches between multiple resource sets. InOpenGL, this corresponds to a single OpenGL context with a shared framebuffer object (FBO) and separate render buffer for each VAR scene.

In the case where it is not feasible to associate a shared resourcecontext with multiple views (like in certain technologies such as Flashon a web page), all of the resources can be loaded into a single objectcorresponding to a view in the respective graphics API that is thenmoved around and drawn to update each specific VAR scene. In lieu ofhaving more than one single active viewport, other viewports can beapproximated by a static representation while they are not active. Whena VAR scene is selected to be actively drawn, the projection functiondefined in Step S170 can interpolate between the static representationand the desired active representation. Any suitable method can be usedfor selecting which VAR scene should be active in a resource set,including but not limited to, touching or clicking a specific VAR sceneor hovering over a specific VAR scene.

Step 160B is executed if a required resource is not loaded, andfunctions to add the resource is added to a queue to be loaded. So as tonot slow down the rendering of the rest of the graphics subsystem, it ispreferable (but not required) to load the required resourceasynchronously in another thread. One implementation of this, of manypossible, involves dispatching a block of work on a grand centraldispatch queue to be executed in a background thread, that shares OpenGLcontext resources with the drawing thread through an OpenGL sharegroup.It may be beneficial to split up resource loading into priorities baseon user demands or estimated load time so that something small, resourcewise, can be drawn sooner than would be possible if the block of work toload it was behind a larger resource request. Practically, this maymanifest itself as rendering low-resolution thumbnails rather than fullresolution imagery in the case of photography. The delegation ofpriority can be delegated to either the producer or consumers of thequeue.

Step 170 involves calculating the projection of the VAR scene using acombination of the global transform parameters, VAR scene context, thelocation of the VAR scene frame within the device frame, and otherglobal context. This preferably involves utilizing either a sharedfunction closure or a function or closure specific to each individualVAR scene.

In the common case, a shared function calculates a perspective matrixfrom the perspective parameters in the global transform parameters andthen premultiplies this resulting matrix by orientation and translationfrom the global transform parameters (representing in a matrixconceptually similar to a View Matrix in OpenGL terminology) to achievethe final projection matrix. Generating the perspective matrix ispreferably done through whichever mathematical model is suitable for agiven rendering technology used in Step S180. With OpenGL, this ispreferably, but not limited to, a function with a mathematicalequivalence to gluPerspective or glFrustum, which take in field of view,aspect ratio, near plane, and far plane or left, top, bottom, right,near and far bounds parameters respectively. These methods typicallyinvolve, but are not required to involve, a transformation to from eyecoordinates to homogeneous clip coordinates in a frustum. Othertechnologies will likely, but are not required to use similarmathematics, altered depending on where in the rendering pipeline theperspective divide occurs (if at all), the clipping model, and the depthprecision or other technology-specific concerns. Any suitablemathematical model for a projection that simulates optical perspectivemay be used.

Other functions may require more parameters than just the globalparameters, such as the position of the VAR scene frame relative to thedevice frame which can be calculated in Step S171 by performingwhichever platform specific methods are needed to calculate the boundsof the view. On iOS platforms one can use methods to traverse the viewhierarchy and get the absolute position, on HTML platforms, one maytraverse the DOM hierarchy which can be combined with scroll offsets tocalculate VAR scene frame relative to the display. Any suitable methodmay be used. Additional VAR scene context may be used, such as, but notlimited to whether the intent of a specific VAR scene is to highlight aspecific direction S173. This extra context may be sourced from memory,disk, network or any other suitable storage medium. Additional globalcontext, may be used in this function S174, such as, but not limited towhether all VAR scenes should be frozen or not. This extra context maybe sourced from memory disk, network or any other suitable storagemedium or sensor input.

Another possible function calculates an additional orientation based onthe position of the VAR scene frame within the device frame which isthen multiplied with the global orientation from the global transformparameters after which a perspective transform computed from the globaltransform parameters is multiplied. This additional orientation may be,but is not limited to being computed as a pitch rotation scaledaccording to the location of the VAR scene frame center along thevertical axis of the display (or in the case of a display without anobvious orientation along the axis pointing away from gravity). When aVAR scene is placed inside a scrollable view, scrolling the view up anddown gives an effect similar to what it would be like to look through alensed window. FIG. 2 depicts how swapping the frames of two VAR sceneswith this function would change the resulting projection. Anotherpossible function, which is more optically accurate, gives a similareffect which skews and modifies the perspective projection instead ofusing a rotation to simulate this effect.

Another possible function adjusts the field of view based on theposition of the VAR scene frame within the device frame which is thenmultiplied with the global orientation from the global transformparameters after which a perspective transform computed from the globaltransform parameters is multiplied. This additional field of viewscaling may be, but is not limited to being computed according to thelocation of the VAR scene frame center along the vertical axis of thedisplay (or in the case of a display without an obvious orientationalong the axis pointing away from gravity). In one of many possibleconfigurations, when a VAR scene is placed inside a scrollable view,scrolling the view up and can give an effect, such as zooming in as theVAR scene center approached the center of the device frame and zoomingout as the VAR scene center moves away from the center of the deviceframe. Alternatively, another method may use the distance from from thedevice to the user's viewing location to monotonically adjust the fieldof view, thereby simulating the effect of looking through a window.

Another possible function could be composed from any of the abovefunctions where the device orientation is replaced with a staticallydefined orientation. This function could be used to highlight a specificportion of the VAR scene by locking the orientation to look in aspecific direction.

Another possible function could be composed from any of the abovefunction where the perspective component of the projection is skewedaccording to the displacement of the user's viewing location relative tothe device.

Another possible function could be composed from any of the abovefunctions with a programmed set of values such that as time progresses,the VAR scene results in a tour through the different aspects of a VARscene.

Another possible function could select and compose any of the abovefunctions depending on the context of the VAR scene being shown. Forexample, a user may deliver some input, such as but not limited to,tapping a button, that switches between a function that freezesorientation to one that uses the global orientation withoutmodification. Such a function could transition between any otherpossible functions.

Step S180 involves using the projection calculated in S170 to render theVAR scene onto a display, framebuffer, or other suitable image storagelocation. This step preferably takes advantage of platform specific APIssuch as OpenGL, Direct X, WebGL to render the 3D scene but any suitablerendering technology which supports perspective projections may be used.Preferably, this may occur on a thread separate from the thread thathandles user interface events, but if a computer processing unit is fastenough more work can be done on the user interface thread.

After step S180 is complete, another VAR scene is chosen from thecollection of requested VAR scenes and processed through steps S150,S160A/B, S170, S171, S173, S174, and S180 until all VAR scenes have beenprocessed at which point the process repeats again from S110.

If the layout of a subset of VAR scenes permits a rendering multiple VARscenes at once (for example, but not limited to, the scenario where aset of VAR scenes are intended to be rendered in a grid), then steps maybe pipelined such that S150 is performed on a batch of VAR scenes beforethose VAR scenes go through Step S160A/B, etc. When possible thisimproves performance by reducing the amount of overhead the CPU wastescommunicating with a graphics API such as OpenGL on the GPU.

Concurrently, two other submethods may be followed. One submethod is theresource loading method, which is designed to ensure consistentperformance of the graphics rendering thread by offloading resourceloading to an asynchronous background queue. This method may preferablybe executed on a dispatch queue, but any other suitable concurrencymechanism such as, threads, processes, green-threads, pre-emptivemultitasking, multi-core or distributed computing may be used.Communication between this submethod and the graphics rendering threadmay occur through any suitable mechanism such as synchronization withlocks or semaphores, lock-free data structures, interprocesscommunication, message passing, shared memory, network sockets or anyother suitable communication mechanism. Resources may be shared usingany suitable memory sharing technology, preferably OpenGL sharegroupswhich allow for two separate OpenGL contexts on different threads toaccess the same set of resources such as textures and other resources.

Step S210 includes waiting for a resource load request to be processed.This can either be done through a queue or any other collectionprimitive. Preferably one may use a first in last out queue (FILO),which ensure that whatever the latest VAR scene to be added to the frameis rendered first. This is improves experience when a user is scrollingpast large set of VAR scenes. When using a first in first out queue(FIFO), the user may have to wait for the intermediate bubbles that areoff screen to finish loading before the VAR scenes currently on screenare loaded. Alternatively, if the queuing mechanism has an efficientmeans to cancel a resource load request, one may keep a FILO queue,certain queing technologies when combined with memory management canmake it difficult to cancel a request, namely grand central dispatch oniOS platforms, without greatly added complexity.

Step S220 involves loading the specified resource, which can be either atexture, list of vertices, faces, triangles, shapes or other resourceused in rendering a VAR scene. When possible, compressed formats whichrequire little CPU decoding can be used to accelerate this process.

An additional submethod which may be required is one that reduces thememory consumed by the rendering method. In many scenarios where swapmemory is to be avoided for performance or when there is no swap memoryat all (such as on many mobile phones and tablets), the operating systemor application may request that it minimizes the required memoryfootprint. This submethod involves collecting all of the VAR scenes,checking for which have loaded resources, optionally checking if theyare not being drawn (and thus are safe to evict without impacting whatis drawn on screen), and evicts each VAR scene's resources. This is caneither be processed in-line with the rest of the drawing methods orasynchronously like resource loading.

Step S310 is identical to Step S140 in collating requested VAR scenesthrough either a global registry or other equivalent means.

Step 320 is identical to Step S160 in checking the implemented resourcemodel for the presence of a specific resource, except in the case thatthe resource is not loaded in which case the next VAR scene isprocessed.

Step 330 is the inverse of Step S150 in that it checks that the VARscene is not visible to the user. By only evicting invisible scenes, asuboptimal user experience from flickering VAR scenes resulting fromquickly loading and unloading resources is avoided. However, thedrawback is that the iteration through each VAR scene may take longerwith this step included and may add complexity. Additionally, theoperating system may require that more memory be released faster andthus it may be optimal to evict all resources and have them lazilyloaded back in as they are requested to be drawn.

Step 340 consists of actually releasing the resources in whichever wayis mandated by the graphics API. In OpenGL, for example, this meansissuing a call to delete a texture

Other implementation details remain around the memory management model.As VAR scene resources can be very large, particularly, when they aretextures, it may be ill-fitting to rely on the garbage collection orbuilt in reference counting schemes. Some reference counting schemes,for example, may delay deallocation or freeing of memory until there arespare CPU resources to do so, as a result, for example, scheduling canresult in a very large resource being allocated before the old resourceis deallocated. As a result, an implementation may, but is not requiredto, implement an additional explicit reference counting scheme which isguaranteed to deallocate a resource when a reference count reaches zerobefore the call to decrement the reference count for the last timereturns.

The end result of this method (and submethods) is an experience in whichit appears that a user interface has multiple ‘live windows’ intodifferent virtual and augmented reality scenes that respond in asynchronized fashion to device movement and user input.

At a higher level of abstraction, this method provides a means todisplay a virtual and augmented reality scene that adjusts its ownprojection based on a set of globally defined parameters andinteractions. As the number of scenes on screen increases this allowsfor easier manipulation as the interaction area of any individualvirtual and augmented reality scene will likely shrink. With globallydefined parameters and interactions, a user is not required toindividually manipulate a virtual and augmented reality scene butinstead can implicitly adjust many scenes shown on the display at oncethrough global transformations on a larger interaction area (forexample, the entire device screen).

The configurable function in Step S170 furthermore allows for anexperience in which, in one possible implementation (of many), the userimplicitly changes the projection of a VAR scene through some othertraditional user interface interaction. One possible configuration is aconfigurable function which takes into account the position of the frameof the VAR scene on the display. As the user scrolls down, implicitlybrowsing through content, the method can be used to implicitly rotateeach VAR scene along a specific axis thereby increasing the amount of aVAR scene the user sees while the VAR scene scrolls along across thescreen as compared to if each projection was fixed. Another example isone such that as a VAR scene approaches the center of a display throughscrolling, the VAR scene zooms in to highlight a specific portion of thescene and then zooms back out as it scrolls again off screen. This hasthe effect of not only showing the context but also highlighting aspecific area by taking advantage of the existing and traditional userinput control of scrolling. Thus, as demonstrated by this configuration,this method may be used to enable the user to experience a dynamic VARscene by building onto existing user interface paradigms withoutrequiring additional interactions, although still allowing them.

A method for dynamically displaying multiple virtual and augmentedreality scenes on a single display of another preferred embodiment mayinclude: determining a set of global transform parameters from acombination of user-defined inputs, user-measured inputs, and deviceorientation and position derived from sensor inputs; calculating aprojection from a configurable function of the global transformparameters, context provided by the user and context specific to avirtual and augmented reality scene; and rendering a virtual andaugmented reality scene with the calculated projection on a subframe ofthe display; and repeating the previous two steps to render additionalvirtual and augmented reality scenes.

The method may additionally or alternatively include wherein detectingorientation from orientation sensor inputs comprises detecting arotation matrix from a gyroscope, magnetometer and/or accelerometer.

The method may additionally or alternatively include whereinuser-defined transform comprises a rotation composed partially of arotation around the gravity vector of the virtual and augmented realityscene. The rotation around the gravity vector may be derived from a userdragging a finger along a vector perpendicular to the gravity vector.

The method may additionally or alternatively include wherein thetransform is partially composed from a zooming gesture used to transformthe field of view. The user's dragging accumulated momentum may continueto adjust the rotation around the gravity vector after the user hasstopped touching the device.

The method may additionally or alternatively include whereinuser-defined orientation comprises a rotation composed partially of arotation defined as the rotation that aligns the gravity vector of thevirtual and augmented reality scene with a vector parallel to thesurface of the device. The vector parallel to the surface of the devicemay be derived from a gesture in which the user drags across the device.The vector parallel to the surface of the device may alternatively bederived from the vector through the center of the display to anotherpoint on a display. The vector parallel to the surface of the device mayalternatively be derived from the vector through the center of a virtualand augmented reality scene frame.

The method may additionally or alternatively include wherein theuser-defined orientation comprises a rotation composed by a yaw andpitch mapped to a cursor location or touch event location along the xand y axis within a subframe of the display. The subframe of the displaymay be the entire display. The subframe of the display may be the frameof the current VAR scene under the cursor.

The method may additionally or alternatively include wherein theuser-measured input consists of the location of the primary user's headrelative to the display. The proximity of the primary user's headrelative to the display may scale the field of view of the projection ofthe virtual and augmented reality scene. The location of the user's headrelative to the display may translate the near plane and far planes ofthe frustum along their respective depth planes. The location of theuser's head relative to the display may rotate the orientation of theprojection along yaw and pitch corresponding to horizontal and verticaldisplacement.

The method may additionally or alternatively include wherein theadditional context is a location of the virtual and augmented realityscene frame within the frame of the device. When the virtual andaugmented reality scene frame is within a scrollable view andcalculating the projection is re-calculated dynamically as the containerscrollable view is scrolled. The location of the center of the framealong the vertical axis may adjust the vertical pitch of the projectionof the virtual and augmented reality scene. The location of the centerof the frame along the vertical axis may scale the field-of-view of theprojection of the virtual and augmented reality scene. The location ofthe center of the frame along the vertical axis skews may translate thenear and far planes of the projection frustum vertically along thebounds of the frame of the virtual and augmented reality scene.

The method may additionally or alternatively include wherein theadditional context provides a fixed projection that causes theprojection calculation to ignore portions of the global transform.

The method may additionally or alternatively include wherein theprojection calculation function is dynamically software defined.

The method may additionally or alternatively include wherein multiplevirtual and augmented reality scenes share one global resource contextbut maintain individual render targets. The rendering of virtual andaugmented reality scenes may occur on separate threads separate fromwhichever thread that handles user input. The rendering of virtual andaugmented reality scenes may maintain separate concurrent queues forloading resources of different resolutions. The rendering of virtual andaugmented reality scenes to the target render buffer may be contingenton the scene frame being within the bounds of the device frame. Whenrequested, the global resource context may reduce its memory consumptionby purging all resources not currently being drawn on-screen. Multiplevirtual and augmented reality scenes may be rendered in one call to thegraphics processor by simulating multiple render targets with a singlerender target and an alpha mask. Individual render targets may berepresented as independent objects subclassing platform specific userinterface components, which are drawn to sequentially individually or inbatches by a globally shared draw loop.

The method may additionally or alternatively include wherein multiplevirtual and augmented reality scene share one global resource contextand share a single render target that is moved throughout a userinterface as drawing is needed. Virtual and augmented reality sceneswhich have projections which are not actively being updated may bereplaced with a static approximation of each respective projection.

An alternative embodiment preferably implements the above methods in acomputer-readable medium storing computer-readable instructions. Theinstructions are preferably executed by computer-executable componentspreferably integrated with a VAR rendering system. The computer-readablemedium may be stored on any suitable computer readable media such asRAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), harddrives, floppy drives, or any suitable device. The computer-executablecomponent is preferably a processor but the instructions mayalternatively or additionally be executed by any suitable dedicatedhardware device.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

We claim:
 1. A method for dynamically displaying multiple virtual andaugmented reality scenes on a single display comprising: determining aset of global transform parameters from a combination of user-definedinputs, user-measured inputs, and device orientation and positionderived from sensor inputs, calculating a projection from a configurablefunction of the global transform parameters, context provided by theuser and context specific to a virtual and augmented reality scene; andrendering a virtual and augmented reality scene with the calculatedprojection on a subframe of the display; and repeating the previous twosteps to render additional virtual and augmented reality scenes
 2. Themethod of claim 1, wherein detecting orientation from orientation sensorinputs comprises detecting a rotation matrix from a gyroscope,magnetometer and/or accelerometer.
 3. The method of claim 2, wherein therotation around the gravity vector is derived from a user dragging afinger along a vector perpendicular to the gravity vector.
 4. The methodof claim 3, wherein the user's dragging accumulated momentum continuesto adjust the rotation around the gravity vector after the user hasstopped touching the device.
 5. The method of claim 1, whereinuser-defined orientation comprises a rotation composed partially of arotation defined as the rotation that aligns the gravity vector of thevirtual and augmented reality scene with a vector parallel to thesurface of the device.
 6. The method of claim 5, wherein the vectorparallel to the surface of the device is derived from a gesture in whichthe user drags across the device.
 7. The method of claim 1, wherein theadditional context is the location of the virtual and augmented realityscene frame within the frame of the device.
 8. The method of claim 7,wherein when the virtual and augmented reality scene frame is within ascrollable view and the second step of claim 1 is re-calculateddynamically as the container scrollable view is scrolled.
 9. The methodof claim 1, wherein the user-measured input consists of the location ofthe primary user's head relative to the display.
 10. The method of claim1, wherein the additional context provides a fixed projection thatcauses the projection calculation to ignore portions of the globaltransform.
 11. The method of claim 1, wherein multiple virtual andaugmented reality scenes share one global resource context but maintainindividual render targets.
 12. The method of claim 11, wherein, whenrequested, the global resource context can reduce its memory consumptionby purging all resources not currently being drawn on-screen.
 13. Themethod of claim 11, wherein multiple virtual and augmented realityscenes can be rendered in one call to the graphics processor bysimulating multiple render targets with a single render target and analpha mask.
 14. The method of claim 11, wherein individual rendertargets are represented as independent objects subclassing platformspecific user interface components, which are drawn to sequentiallyindividually or in batches by a globally shared draw loop.
 15. Themethod of claim 1, wherein multiple virtual and augmented reality sceneshare one global resource context and share a single render target thatis moved throughout a user interface as drawing is needed.
 16. Themethod of claim 1, wherein virtual and augmented reality scenes whichhave projections which are not actively being updated are replaced witha static approximation of each respective projection.
 17. The method ofclaim 11, wherein the rendering of virtual and augmented reality scenesoccurs on separate threads separate from whichever thread that handlesuser input.
 18. The method of claim 17, wherein the rendering of virtualand augmented reality scenes maintains separate concurrent queues forloading resources of different resolutions.
 19. The method of claim 17,wherein the rendering of virtual and augmented reality scenes to thetarget render buffer is contingent on the scene frame being within thebounds of the device frame.
 20. The method of claim 1, wherein thetransform is partially composed from a zooming gesture used to transformthe field of view.